Engineering quantum anomalous Hall phases with orbital and spin degrees of freedom 
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Combining tight-binding models and first principles calculations, we investigate the quantum 
anomalous Hall (QAH) effect induced by intrinsic spin-orbit coupling (SOC) in buckled honeycomb 
lattice with sp orbitals in an external exchange field. Detailed analysis reveals that nontrivial 
topological properties can arise utilizing not only spin but also orbital degrees of freedom in the 
strong SOC limit, when the bands acquire non-zero Chern numbers upon undergoing the so-called 
orbital purification. As a prototype of a buckled honeycomb lattice with strong SOC we choose the 
Bi(lll) bilayer, analyzing its topological properties in detail. In particular, we show the emergence 
of several QAH phases upon spin exchange of the Chern numbers as a function of SOC strength and 
magnitude of the exchange field. Interestingly, we observe that in one of such phases, namely, in 
the quantum spin Chern insulator phase, the quantized charge and spin Hall conductivities co-exist. 
We consider the possibility of tuning the SOC strength in Bi bilayer via alloying with isoelectronic 
Sb, and speculate that exotic properties could be expected in such an alloyed system owing to 
the competition of the topological properties of its constituents. Finally, we demonstrate that 3d 
dopants can be used to induce a sizeable exchange field in Bi(f 1 1) bilayer, resulting in non-trivial 
Chern insulator properties. 



I. INTRODUCTION 

In recent years, dissipationless charge and spin trans- 
port phenomena have drawn very intensive attention. Af- 
ter the topological nature of the anomalous Hall effect 
(AHE) in ferromagnets was discovered and understood^ 
the existence of the spin Hall^ and quantum spin Hall 
(QSH) effects^ was theoretically proposed and exper- 
imentally demonstrated. Nevertheless, the experimen- 
tal observation of the so-called quantum anomalous Hall 
effect (QAHE)j^ characterized by a quantization of the 
anomalous Hall conductivity, is still missing, despite vari- 
ous theoretical proposals^— and recent experimental ad- 
vances in this direction^ 

Numerous studies on the QSH effect have spawned a 
fascinating field focusing on the topologically nontriv- 
ial states of matter, in particular, topological insulators 
(TIs) i 13 ' 14 Remarkable physical properties of TIs orig- 
inate from the gapless surface states, guaranteed by a 
finite bulk gap and time-reversal symmetry. It is plau- 
sible to seek QAH effects by introducing time-reversal 
broken perturbations to TIs while keeping the topolog- 
ical nontriviality, e.g. via a proximity effect^ Among 
all known TIs^ two-dimensional (2D) systems are of 
particular interest, such as graphene^ HgTe quantum 
wells j£ AlSb/InAs/GaSb quantum wells silicenejil and 
Bi(fff) bilayers (BLs)j 18 i 19 In the past, a lot of work 
has been done based on the extended Kanc-Mele modcl^ 
in the context of graphene^Sr— In these studies, only 
the a band of the p z orbitals arc considered, with key 
terms to induce topological nontriviality being effective 
projected spin-orbit and Rashba spin-orbit interactions, 
arising from second-order perturbative processes^ The 
strength of intrinsic atomic spin-orbit coupling of car- 
bon is limited, though it can be enhanced in fine-tuned 
systems^ 



In the quantized case of the AHE in two dimensions, 
the value of the anomalous Hall conductivity (AHC) is 
proportional to the so-called (first) Chern number, ob- 
tained as an integral of the Berry curvature of the oc- 
cupied states in the Brillouin zone. This integer Chern 
number, which characterizes the "phase" complexity of 
the manifold of Bloch electrons as a whole, has its roots 
in solid state physics since the discovery of the quantum 
Hall effect and its theoretical understanding by Thou- 
less et at in 1982*22, Shortly after that, Haldane sug- 
gested a spinless model to realize the QAHE without 
an external magnetic field^ an effort, which conceptu- 
ally led to the theoretical suggestion of graphene as a 
first topological insulator^ In the latter case, the Chern 
numbers of spin-up and spin-down electrons are non-zero 
due to a microscopic realization of the Haldane's model, 
but they arc opposite to each other. The generality of 
this counteracting nature of the Chern numbers in TIs 
lead eventually to the concept of the spin Chern num- 
bers, which can be used to characterize the TI phases 
in time-reversal broken situations^ Ultimately, the re- 
alization of the QAHE is hinged on the presence of points 
in the electronic structure, at which otherwise separate 
spin bands interact with each other, and exchange cor- 
responding spin Chern numbers, to give a global non- 
vanishing Chern number in the system. The mechanism 
behind such a Chern number exchange is strongly de- 
pendent on the details of symmetry, hybridization and 
microscopic structure of the spin-orbit interaction in the 
system, making search for systems which could exhibit 
non-zero QAHE difficult, yet definitely rewarding since 
the physics involved is possibly very rich. 

Among other materials, Bismuth (111) bilayer (BL) 
has been shown to host nontrivial topological properties 
due to the strong SOC of Bi i 18 ' 19 i 29 In our previous work, 
we used the concept of the spin Chern number to charac- 
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terize the topological phase transitions in Bi(lll) bilay- 
ers with respect to the strength of an external exchange 
field.— Here, we extend this work further, and demon- 
strate the microscopic origin of the QAH phases found in 
Bi(lll) bilayers when utilizing both orbital and spin de- 
grees of freedom. Based on a general tight-binding model 
for buckled honeycomb lattices with sp-orbitals, we illus- 
trate different Haldane-inspired 6 - mechanisms which can 
lead to a nonzero Chcrn number in each individual band 
upon employing the spin and orbital complexity (sec- 
tion II). We find that in addition to p z orbital based 
physics, in the strong SOC limit, orbital angular momen- 
tum purification can also lead to nontrivial topological 
properties. We further clarify the ways behind achiev- 
ing the QAHE at the so-called spin-mixing points in the 
electronic structure. Finally, using first-principles tech- 
niques, we show that various topological phases occur in 
a realization of a buckled sp honeycomb model — the 
Bi(lll) bilayer — with respect to the SOC strength and 
magnitude of an external exchange field (section III). In 
particular wc suggest the existence of a so-called quan- 
tum spin Chern insulator phase, which is characterized 
by the quantization of both the transverse charge and 
spin conductivities. Wc conclude by discussing how to 
realize the tuning of SOC strength by alloying Bi with 
Sb, and the effect of an exchange field by introducing 
magnetic substrates and/or dopants. 

II. MODEL ANALYSIS 

To illustrate how nonzero Chern numbers and non- 
trivial QAH phases can arise, we first consider a generic 
tight-binding Hamiltonian of sp-orbitals on a buckled 
honeycomb lattice in x-y plane: 

H = J2 U ^ C i^ + J2 C i^ 6il + Bs ^ Ci + H SOC (1) 

id « 

where the first term presents the kinetic hopping with 
(*>J = s iPx-,Py,Pz) as the hopping parameters. The 
second term reflects an orbital-dependent on-site energy 
€i and the interaction with the Zeeman exchange field 
B directed along the z-axis, with I (s z ) as the identity 
(Pauli) matrix. The third term in Hamiltonian ([1} is 
the on-site SOC Hamiltonian. In order to facilitate the 
analysis, s, p z , and {p x ,p y } bands are artificially sepa- 
rated from each other by imposing a rigid shift of e,;. To 
identify different origins of the QAHE, the spin-orbit in- 
teraction is further decomposed into spin-conserving and 
spin-flip parts: 

ffsoc = £l-s = + e(J + *~+J~* + )/2 (2) 

where 1 (s) is the orbital (spin) angular momentum op- 
erator, and £ is the electron-shell averaged atomic SOC 
strength. Since in this work we choose the direction of 
the spin-polarization to be aligned along z direction, the 
spin conserving part of the SOC Hamiltonian, £/ 2 s z , cou- 
ples {p x ,p v } orbitals, while the spin flip part of Eq. [21 



£,{l + s~ + l~s + )/2, couples p z and {p x ,p y } orbitals via 
a flip of spin and a ±1 change in the orbital quantum 
number. Further, we take the Slater-Koster tight-binding 
parameters of Bi, provided in Ref. [3(| for the values of 
hopping integrals of the model, t^j. 

Before proceeding further with the analysis of Eq. ([lj , 
we would like to remind the reader of the Haldanc 
model,' 6 - which is a generic model exhibiting an emer- 
gence of the QAH effect on a honeycomb lattice without 
an external magnetic field. The key mechanism in this 
model is the complex hopping between the next near- 
est neighbor (NNN) orbitals, which leads to a sublattice- 
dependent "magnetic field" with zero flux through the 
overall hexagonal plaquettc. In our work, it is the on- 
site atomic SOC that induces such complex hopping via 
different processes as discussed below. 



A. p z orbitals 

First, we consider the case of the p z bands, partic- 
ularly relevant in graphene physics, well-separated from 
{p x ,Py} and s states. Generally, there are two ways to in- 
duce nontrivial Chern numbers in the p z bands via gener- 
ating an effective next nearest neighbor (NNN) complex 
hopping due to intrinsic atomic SOC. First, on a buck- 
led honeycomb lattice, p z orbitals can hybridize directly 
with the {px,Py} orbitals, and complex hoppings can be 
induced via the spin-conserving part of SOC which acts 
between p x and p y states. As illustrated in Fig. QJa), 
in this mechanism the corresponding virtual transitions 
readi^i 

l^t)^bLt>^|^t}^l^t) (3) 

where V indicates the direct hopping between p z and 
Px, y orbitals on the neighboring sites, while superscripts 
A and B denote the nearest neighbor atomic sites in sub- 
lattice A and B. 

For the resulting NNN hopping we obtain £nnn ~ £■ 
Importantly, the effective hoppings are opposite in sign 
for p z orbitals of a given spin on A and B sublattice, 
hence giving rise to a finite gap Ai ~ 2£ at the K (K 7 ) 
high-symmetry points in the Brillouin zone, as shown in 
Fig- [Ha), in which a small magnetic field was applied in 
order to separate the spin-up and spin-down bands in en- 
ergy. The gap is topologically nontrivial, in direct accor- 
dance to Haldane's arguments^ The two spin-channels 
here are independent, since only the spin-conserving part 
of SOC is involved, although the complex NNN hoppings 
for the spin-down and spin-up electrons are complex con- 
jugate of each other <2I This means that the Chern num- 
bers of the spin- up bands are opposite to the Chern num- 
bers of the spin-down bands, as shown in Fig.QJa), which 
suppresses the QAHE of occupied electrons. 

Secondly, on-site spin-flip SOC can give rise to complex 
next neighbor hopping too, even if there is no direct hy- 
bridization between p z and {p x ,p y } orbitals, Fig.QJb). In 
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FIG. 1: Topological analysis of p z ((a)-(c)), s ((d)-(f)), and {p x ,p y } ((g)-(i)) bands in a buckled honeycomb bilayer. The 
electronic structure is obtained using the tight-binding parameters for bulk Bi as listed in Ref. [33] , with an additional artificial 
rigid shift of on-site energies yielding separation of s, p 2 , {p x ,p y } states in energy. Only nearest neighbor (NN) hoppings 
are considered, with V VV7t = (this does not affect our conclusions, see the main text). Without specification, the spin-orbit 
coupling parameter £ = 4.5 eV and exchange field B = 0.1 eV are used. Left column (a,d,g) (middle column (b,e,h)) corresponds 
to the case with only spin-conserving SOC (spin-flip SOC) included, while the full SOC is considered in the right column (c,f,i). 
Red (blue) in (a-f) and upper inset of (i) stand for the spin-down (spin-up) states, while in (g-i) and lower inset of (i) for the 
states with orbital magnetic number m = +1 (m = —1). Dashed horizontal lines indicate the Fermi energy level adjusted 
such that relevant bands (for instance, p z in (a-c)) are half-filled. Numbers denote the Chern number of each individual band. 
Insets in (a-g) ((i)) display the electronic structure at the Dirac critical point at ±0.01 eV (±0.06 eV) with respect to the 
Fermi energy, and sketches illustrate different channels for complex nearest neighbor hopping (red (yellow) circles denote p z 
(s) orbitals, while p XtV orbitals are indicated by blue ellipsoids; black (red) arrows depicts £nn hoppings (SOC hybridization), 
respectively.). Text below each panel denotes the hybridization of {p x ,p y } orbitals to s and p z orbitals by kinetic hoppings, 
where ± ( — ) indicates (no) hybridization. For instance, "s — {p x ,p y } ±p 2 " in (a),(i) means that there is no hybridization 
between s and {p x ,p y }, while {p x ,p y } and p z are coupled. 
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this case, the corresponding virtual transitions are : 25 ' 31 i 32 

I Pt t) P ty |) ^| Ply |) ^| Piy |) P t t) 

(4) 

where £fn p = £(l + s + I s + )/2, and V stands for the 
direct hybridization between p xy orbitals on neighbor- 
ing A and B sites. In analogy to the case with spin- 
conserving SOC considered previously, the effective hop- 
pings within A and B sublattices are of opposite sign 
and the resulting gap is also topologically nontrivial. In 
this case £nnn ~ £ 2 , and the corresponding gap A2 at K 
(K 7 ) (Fig. QJb)) scales quadratically with respect to the 
SOC strength. If there is no hybridization between the p z 
states of the opposite spin as what occurs in the 2D pla- 
nar honeycomb lattice, this second situation without con- 
sidering an external exchange field actually presents the 
topological insulator state as found in graphene^ consist- 
ing of two independent spin copies of the Haldanc model. 
Thus, the resulted QAH conductivity for the half-filled 
situation is again zero, although the p z bands of opposite 
spin are meeting at the Fermi energy. 

Note that, while the arising non-zero Chern numbers 
of the bands in all situations discussed in Fig. 1 could 
lead to a QAH effect upon inducing a very large (larger 
than the bandwidth) exchange splitting in the system, 
we do not discuss such "trivial" possibility here. Thus, 
to induce nontrivial QAHE in the p z bands, the spin-up 
and spin-down p z bands have to be coupled, inducing ex- 
change of the Chern numbers. This can be achieved by 
allowing for the hybridization between p z and {p x ,Py} or- 
bitals and upon taking the complete SOC into consider- 
ation. When two entangled bands of different spin char- 
acter from conduction and valence bands overlap with 
each other, spin-mixing occurs and the Chern number 
exchange takes place when the gap is opened, Fig. HJc). 
For example, for p z orbitals, a possible transition process 
is: 

I P A t)^ p \p A , y D^'IpU)- (5) 

The magnitude of the gap, opened between up and down 
Pz-bands, as shown in Fig. HJc), is proportional to £ 2 
according to our calculations. 

The variation of Chern numbers upon band touching 
is determined by the Berry indices at degenerate points 
in a generalized parameter space (k, 77), which is defined 

IndxBerry = — [ d,S il ■ II, (6) 

with S 2 as a two dimensional surface enclosing the degen- 
eracy point, n is the surface normal vector pointing out- 
wards, f2 is the Berry curvature.— In our case, rj can be 
the strength of the exchange fields or SOC. Since S 2 can 
be chosen arbitrarily small, the magnitude of IndxBerry 
can be obtained by examining the band dispersion at the 
degeneracy point i 33 ' 34 For instance, if the dispersion of 
the crossing bands is linear (quadratic), the variation of 



the Chern number is one (two) . This saves us from evalu- 
ating the Chern number exchange explicitly, although the 
sign of IndxBerry cannot be determined from this simple 
argument. In the case of Fig.QJc), the Chern number of 
the upper valence band is reduced by 2, while the Chern 
number of the lower conduction band is increased by 2, 
as compared to the situation in Figs.QJa) and (b), when 
the hybridization between the p z bands of opposite spins 
is introduced. The reason for this is that, although the 
dispersion of the bands at the point of degeneracy is ob- 
viously linear, Fig. [ljc), there are two such points in the 
Brillouin zone (K and K'). 

In previous studies of the QAHE in the context of 
graphenepSr— an extended Kane-Mele model^ was em- 
ployed with an additional Rashba term. The effective 
Rashba spin-orbit interaction originates from the combi- 
nation of intrinsic SOC and potential gradient perpen- 
dicular to the honeycomb lattice plane which breaks the 
inversion symmetry, and can be created by e.g. apply- 
ing a finite electric field perpendicular to the honeycomb 
plane 2 ^ or imposing a finite curvature of the honeycomb 
sheets In this case, when the time-reversal symmetry is 
broken via, e.g., non-zero magnetization due to adatoms, 
the QAHE can be induced.—^ Nevertheless, the effective 
complex hoppings due to the Rashba spin-orbit arise be- 
tween p z orbitals of opposite spin on the nearest neighbor 
sites — a situation, topologically distinct from the com- 
plex NNN hoppings considered in this work. 



B. s orbitals 

The second-order perturbation processes discussed 
above can also take place if the s bands are taken in- 
stead of p z states. It is achieved by hybridization between 
s and p orbitals, Fig. [Tf d)-(f) . For instance, the effec- 
tive complex hoppings between s orbitals can be induced 
by spin-conserving SOC via hybridizing with {p x ,p y } or- 
bitals (Fig.QId)) or spin-flip SOC (Fig^e)). Comparing 
to the p z orbitals, the only difference is that for the case 
with spin-flip SOC (Fig. [He)), the corresponding virtual 
transitions are 

I S A t) ^| Ply t) ^1 Pi I) ^\ Ply t) ^1 S A t). 

(7) 

Note that the role of | p® y ) and | p^} in this case can 
be exchanged, i.e., electrons can hop from | s A ) to | pf ) 
and then couple with | p^ ). Furthermore, similarly to 
the cases for p z bands (Fig.fTJa-c)), the spin-up and spin- 
down s bands are decoupled with Chern numbers of op- 
posite sign (Fig. [ijd) and (e)), and a nontrivial QAH 
effect for the half-filled case takes place only when the 
two spin-channels are entangled, i.e., due to hybridiza- 
tion with p z orbitals (Fig. Qlf)). 
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C. {Px,p y } orbitals: orbital purification 

More interestingly, nontrivial topological properties 
arise in the {p x ,p y } bands as well, Fig.QJg-i). If only the 
spin-conserving SOC is considered, every {p x ,p y } band 
acquires a nonzero Chern number when the magnitude 
of SOC is larger than a critical value £ op (Fig. QJg)). 
This is different from the cases considered for p z and s 
bands, where the strength of SOC determines only the 
size of the gap, while it is irrelevant for the topological 
properties of individual bands. Our analysis reveals that 
the origin of the nontrivial Chern numbers can be at- 
tributed to the so-called orbital purification, caused by 
strong SOC (Fig.[IJg)). For instance, for the middle four 
bands in Fig. [TJg), the Chern numbers are nonzero only 
when SOC is larger than £ op = 5.4 eV, when the value of 
the orbital angular momentum L z in each of the bands 
becomes consistently either strictly positive or negative 
at each k point (see the coloring of the bands in Fig. [TJg)). 
This leads to the predominantly m = ±1 (p x ±ip y ) char- 
acter of each band, which constitutes the essence of or- 
bital purification. The complex NNN hoppings in this 
case are induced via the following mechanism: 



p^± t p^)^\p*Tip B y ) 



B\ V 



P x ± iPy) 



(8) 



due to virtual transitions between the bands of differ- 
ent m on different sublatticcs, mediated by kinetic hop- 
ping. Since only spin-conserving SOC is considered here, 
the spin-up and spin-down channels are not entangled, if 
no direct hybridization of {p x ,p y } and other orbitals is 
present, which corresponds to the situation discussed by 
W u 36 i 37 for spinless honeycomb optical lattices. 

On the other hand, for the lowest (top most) two 
bands, the orbital angular momentum purification occurs 
for an even smaller SOC strength of about 0.3 eV, when 
only spin-conserving SOC is considered. There exists also 
another possibility for these bands to acquire a complex 
NNN hoppings by spin- flip SOC as shown in Fig.[TJh). In 
this case, the Chern numbers for the middle four bands 
are always zero even if the spin-flip SOC strength is in- 
creased to very large values, while for the lowest (top 
most) two bands, the Chern numbers are quantized with 
nonzero value. To obtain a nontrivial QAHE, entangle- 
ment between two spin channels is again essential, where 
Chern number exchange occurs via spin-mixing as ob- 
served for the s and p z cases above. For {p x ,p y } orbitals, 
such entanglement can be achieved by hybridization with 
p z -orbitals, with corresponding Chern number exchange 
of 1 due to a linear band dispersion at the points of de- 
generacy. Following previous arguments, this leads to a 
non-zero QAH conductivity with C = — 1 at half- filling, 

Fig. mi). 

The orbital purification is determined by the compe- 
tition between the strength of the on-site SOC and the 
band width due to kinetic hoppings. In the model anal- 
ysis above, we neglected the V pp7r hoppings in Bi, with 
resulting £ op of about 5.4 eV. This critical value is sig- 



nificantly reduced if V pp7r hoppings are taken into ac- 
count, which reduces the band width of the p bands. In 
the latter case, for Bi, £ op becomes 2.4 eV if all relevant 
hopping integrals for the bilayers, as listed in Ref. [30| . 
are included in our tight-binding calculations. Our first 
principles calculations of the Bi (Sb) bilaycr, on the other 
hand, show that the magnitude of the atomic SOC for Bi, 
?Bi (£sb for Sb), is about 2.8 eV (0.9 eV). Thus, Bi bilayer 
resides in the strong SOC limit, where the orbital purifi- 
cation takes place, orbitally polarized p x ± ip y bands are 
formed and exhibit non-zero Chern numbers. To verify 
this explicitly, we evaluate the expectation value of the 
orbital momentum operator L z in Bi bands. As shown 
in Fig. [21 the first and the third valence bands of Bi(lll) 
bilayer are indeed orbitally purified in the entire Brillouin 
zone. The corresponding Chern numbers are ±2 and ±1, 
respectively. The orbital purification in these bands oc- 
curs until the SOC strength of Bi atoms is scaled down 
to 70% of its atomic value, that is, approximately 2 eV, 
when the Bi bilayer looses its topological insulator prop- 
erties, see Fig. 3(a). In this sense, we can call the Bi 
bilayer an oi~bital topological insulator, as opposed to the 
case of graphene, in which all the topological properties 
are due to p z states. 



3.0 
2.0 
1.0 



> 

& 0.0 

Ml 

s -i-o 
w 

-2.0 
-3.0 
-4.0 



5 

2 



A 



V 



M K 



K' 



FIG. 2: Orbital angular momentum L z purification in Bi(f If) 
bilayer from first principles calculations. Red (blue) stands for 
the expectation value of the L z operator. A small artificial 
exchange field of 10 meV was applied perturbatively to split 
originally degenerate bands. In each pair of resulting nearly 
degenerate bands the expectation value of L z , s z and the 
Chern numbers are opposite to each other. Integers next to 
the bands stand for the absolute value of the Chern numbers. 



A comment on achieving nontrivial topological phases 
in optical lattices is in order, given that the idea of or- 
bital angular moment purification was first proposed in 
the context of ultracold atoms^ Quite recently, both 
Abelian^ and non-Abelian 3 ^— gauge fields (see Ref. [42j 
for a recent review) have been experimentally realized 
for trapped neutral atoms. Since ultracold atoms are 
spinless particles, in this context the relevance of the 
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models discussed here is limited to the cases shown in 
Fig. QJa),(d),(g) when neglecting the spin degree of free- 
dom. Moreover, the hybridization between different or- 
bitals (as in Fig. [TJa) and (d)) is not necessary, since in 
optical lattices complex hoppings are carried by dressed 
states. Experimentally, ultracold atoms in higher orbitals 
such as p bands have been recently achieved (sec e.g. 
Ref. [H[), which makes them a promising candidate for 
realizing various topological properties. 

To summarize, we demonstrated that both orbital and 
spin degrees of freedom can be utilized to induce nonzero 
Chern numbers in an individual band, while nontrivial 
QAH effect is due to the entanglement of two spin chan- 
nels accompanied by spin exchange of the Chern num- 
ber. In real materials, all bands are in general coupled 
by hybridization and, hence, Chern number exchange is 
expected at every non-accidental band crossing. In this 
sense, graphene is a peculiar material^ since the well- 
separated in energy a and 7r bands are not coupled by 
kinetic hopping due to graphene's planar structure. Sub- 
lattice staggering as found in siliceneil is too small to 
induce significant variation of the electronic structure, 
thus the simplified 4-band model considering only the 
7r bands is good enough to account for the topological 
phase transitions in graphene-related systems How- 
ever, for Bi(lll) bilayer and its derivatives, all three p- 
orbitals reside in the same energy scale, the hybridiza- 
tion between them is strongly enhanced due to buckling, 
and the strength of SOC is orders of magnitude larger as 
compared to that of carbon or silicon atoms. Therefore, 
rich physics with competing orbital and spin degrees of 
freedom is expected in the latter case. In the next sec- 
tion, we will demonstrate that non-trivial QAH phases 
can be achieved in Bi(lll) BL by applying an exchange 
field, where the variation of Chern numbers can be ex- 
plained using the Chern number exchange scheme illus- 
trated above. 



III. TOPOLOGICAL STATES OF Bi AND Sb 
BILAYERS 



generalized to the cases with spin-flip SOC by Prodan^ 
Compared to the more universal approach of Ref. j47j . 
the spin Chern number can be easily constructed. It is 
applicable to the situation without time-reversal symme- 
try, and robust when the spectra of Ps z P are gapped in 
the BZ, where P is the projection operator onto the oc- 
cupied states. In the following, we use C+ (C_) to denote 
the "Chern" number of the spin "up" ("down") projected 
occupied bands, and the spin Chern number is defined as 
C s = ±(C+ - C_). For more details see Ref. 

In this section we tackle the question of finding and 
characterizing possible topological phases of Bi and Sb bi- 
layers from first principles, as the corresponding Hamilto- 
nian of these systems is altered by scaling the SOC matrix 
elements and applying an external exchange field. Our ab 
initio calculations were performed using the full-potential 
linearized augmented plane wave method as implemented 
in the Jiilich density functional theory code FLEUR,4£ The 
Wannicr functions technique was used on top of self- 
consistent first principles calculations to derive an accu- 
rate tight-binding Hamiltonian of the system42r— The 
relaxed bulk in-plane lattice constant and the distance 
between the two layers for Sb(lll) constitute 4.30 A and 
1.55 A, respectively, while for Bi(lll) bilayer the corre- 
sponding values are 4.52 A and 1.67 A. The details of 
calculating the anomalous (spin) Hall conductivities, are 
described in Ref. (29|. Moreover, we find that the spec- 
tra of s z projected onto the occupied states are always 
globally gapped for the cases considered in the following, 
despite the fact that the strength of SOC in Bi, is 
very strong (« 2.8 eV), leading to well-defined spin Chern 
numbers. A uniform exchange field was applied on top 
of the first principles electronic structure, as described in 
Ref. [2j|. The spin-orbit strength £ in our calculations 
was scaled by hand via multiplying all atomic SOC ma- 
trix elements with a uniform scaling parameter during 
the self-consistency cycle. In order to clarify the effect of 
a more realistic exchange field, we have performed addi- 
tional calculations of a Bi(lll) bilayer on ferromagnetic 
substrates (europium chalcogcnidcs) and in the presence 
of magnetic dopants in the system, as described later. 



In the previous section we illustrated within a tight- 
binding model how non-zero Chern numbers can arise 
due to intrinsic SOC and how to induce the QAH effect 
via applying an exchange field. One problem remains 
unsolved, however, that is the problem of characterizing 
various topological phases consistently. For instance, TIs 
are characterized by the Z2 indcx^ and QAH insulators 
by the (first) Chern number, and it is still not completely 
clear how to properly characterize the 2D insulating topo- 
logical phases which do not fit into the these two classes, 
e.g. insulators with broken time-reversal symmetry which 
originate from TIs and have a zero Chern number. An- 
other issue is how such "intermediate" phases can be 
probed and distinguished experimentally from topolog- 
ical and Chern insulators. To this end, we use the spin 
Chern number first introduced by Sheng et a/4£ and later 



A. Phase diagram of Bi(lll) bilayer 

The calculated phase diagram of the Bi(lll) BL with 
respect to the strength of atomic SOC, £, and the magni- 
tude of the exchange field, B, is shown in Fig.^a). The 
emerging distinct topological phases can be characterized 
by the value of the AHC (which equals the Chern number 
times e 2 /h in the insulating regime) and the spin Chern 
number. The topological insulating phases are separated 
by a metallic phase, with topological phase transitions 
occurring during closing the bulk band gap and reopen- 
ing it again as B and the SOC strength are varied. As 
confirmed by the calculation of the spin Chern number, 
when the time-reversal symmetry is not broken (B = 
cV), the Bi(lll) BL is a trivial insulator (C s = 0, C = 0) 
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FIG. 3: Quantum anomalous Hall phases in Bi/Sb(lll) bilayers, and electronic structure of the Bi(lll) bilayer on magnetic 
substrates and with magnetic dopants, (a) ((c)) displays the phase diagram of the Bi(lll) (Sb(lll)) bilayer with respect to the 
strength of atomic SOC and magnitude of exchange field B. Numbers denote the Chern number in the QAH phase, "TI" stands 
for the TI phase, while "TRBTI" stands for the time-reversal broken TI phase. The horizontal dashed line in (a) indicates the 
case with B = 0.5 eV, for which the AHC and spin Hall conductivity (SHC) are shown in (b) with respect to the strength of 
SOC. Inset in (a) displays the dispersion of the edge states (red lines) and the projected bulk states (gray shaded region) in 
Bi(lll) BL ribbon with B = 1 eV and 70% of the Bi atomic SOC strength in the C = +1 phase, while that in (c) displays the 
dispersion of the edge states in Sb(lll) BL ribbon with B = 1 eV and 100% of the atomic SOC of Sb in C — —1 phase, (d) 
displays the band structure of the Bi(lll) BL on top of EuSe(lll) terminated with Se atoms. Black (red) lines denotes the 
majority (minority) bands. The exchange splitting at K point for the first two pairs of the valence bands is about 60 meV. The 
dashed horizontal line indicates Ef, and the inset displays the positions of the atoms. Left panel of (e) shows the electronic 
structures of Bi(lll) BL in 2 x 2 superstructure with one Fe atom located at the hollow site in between the two Bi layers, 
whereas AHC conductivity is shown in the right panel. The horizontal dashed line in (d) indicates the position of the Fermi 
level which is not in the gap due to the fact that Se assimilate electrons. Shaded region in (e) marks the gap with the Chern 
number C = +2. (f) is analogous to (e), but with the SOC strength of Bi scaled down to 20% of its atomic value. 



for the SOC strength < 56% of its atomic Bi value, while 
it turns into a TI (C s = — 1, C = 0) when £ is larger than 
67% of £ B i^ In the TI phase, C+ = -C_ = -1, resulting 
in a spin Chern number of —1. 

Breaking the time-reversal symmetry by applying an 
exchange held induces topological phase transitions; If 
the magnitude of the induced exchange splitting is large 
enough to make the two bands, which were originally 
below and above the gap, to overlap, according to the 
mechanism depicted in Fig. QJc),(f),(i). Before that, the 
original insulating phase is not destroyed, and to a certain 
extent preserves its topological properties in the presence 
of a small exchange field. For example, for 0.2 £ei < £ < 
0.56 £bi the trivial insulator phase at B = retains as the 
B is increased until the system enters the metallic phase 



in which the originally separated bands directly overlap. 
On the other hand, for Bi BL with £ > 0.67 £ei, the 
original topologically nontrivial insulating gap is finite for 
finite values of B, with C+ = — C_ = — 1 until entering the 
metallic region. The resulting spin Chern number is in 
this case C s = — 1, which manifests the occurrence of the 
time-reversal broken TI (TRBTI) phase, observed also 
for graphene in (52|, and discussed at length in e.g. [29| . 

The transition into the QAH phase upon increasing the 
B always takes place via an intermediate metallic phase, 
Fig. EUa). In our previous work we have shown the ap- 
pearance of a QAH phase with C = — 2 for Bi BL with 
full atomic SOC and B > 0.42 eV~2- As we can see from 
Fig.^a), in a Bi BL with intermediate SOC strength, ap- 
plying strong exchange field can also induce QAH phases 
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which arc derived from either a metallic, trivial insu- 
lator, or topological insulating phases at B = 0. For 
instance, in a Bi BL with SOC strength scaled to one 
half of the atomic value, a QAH phase with C = +1 
emerges for B > 0.25 eV. It is an exotic phase, in which 
C- = +1 while C+ = 0, as compared to the QAH phase 
with C = —2, in which C_ = C+ = — 1. In the former 
case, the spin Chern number C s = ^(0+ — C_) is not an 
integer, and there exist only one chiral edge state local- 
ized at each edge of a one-dimensional Bi(lll) ribbon, 
as shown in the inset of Fig.[3Ja). Our calculations show 
that these edge states are spin-polarized in the same di- 
rection, leading to the quantized value of the AHC (in 
units of +e 2 /h) and also finite and large values of the 
SHC, as demonstrated explicitly by calculations for the 
case of B = 0.5 cV in Fig. G^b). 

The peculiarity of the topologically nontrivial phases 
with nonzero C s lies in their finite SHC. It can be under- 
stood in an intuitive way following two equations below: 

AHC = ff f + 

(9) 

SHC = <j t -a l 

where (a^) denotes the conductivity of the majority 
(minority) electrons in units of e 2 /h (e/Air) for the AHC 
(SHC), respectively. These relations can be defined and 
hold true only when the spin-flip band transitions due 
to spin-non-conscrving part of SOC are absent. In the 
trivial insulator phase, both the AHC and SHC are zero 
since C+ = C_ = 0. When time-reversal symmetry is not 
broken, i.e. for the Bi BL in the TI phase, it implies that 
cr = —cr = — 1, leading to zero AHC and to a quantized 
SHC of -2e/47rJ£ Nevertheless, due to the spin-flip part 
of SOC, the SHC of the Bi BL in the TI phase is reduced 
to about 0.7 e/47r, as we found in our previous work 
(Fig. 3 in Ref. [29j). In the QAH phase with C = -2, 
C+ = C_ = —1, and, correspondingly, a* = cr, leading 
to zero SHC without the spin-flip transitions. However, 
the exchange splitting in the system leads to the fact that 
the spin-flip scattering between majority and minority 
states is not balanced, leading to a small but finite SHC, 
as shown in Fig. G^b) for Bi BL at B = 0.5 cV. 

On the contrary, for the QAH phase with the Chern 
number C = +1, the majority spin channel is switched off 
since C+ = 0, and the resulting SHC is enhanced in mag- 
nitude as compared to the QAH phase with C = — 2, see 
Fig. EJb)- As we artificially suppress the spin-flip band 
transitions by switching off the spin-non-conscrving part 
of SOC in our calculations, we observe that the value of 
the SHC acquires a quantized value of the magnitude of 
— (not shown), despite the fact that the area of the 
C = +1 phase shrinks as the electronic structure of the 
bilayer is modified when the spin-flip SOC is switched off. 
That is, we demonstrated that it is possible to achieve 
a coexistence of the QAHE and quantized (in the spin- 
conserving sense) spin Hall effect by applying a strong 
exchange field (at constant SOC strength) to a topologi- 
cal insulator, trivial insulator, or even a metal at B = 0. 



We call such an emergent topological phase a quantum 
spin Chern insulator phase. 

The variation of the Chern numbers in the phase dia- 
gram of the Bi BL can be explained by the Chern number 
exchange mechanism at the critical points, as discussed 
in the previous section^ For phase transitions at con- 
stant £ and varying B the Chern numbers are changed 
by +1 (for intermediate £) and —2 (for large £), Fig.[3][a), 
corresponding to linear and quadratic dispersions at the 
critical points, respectively, as confirmed by our calcula- 
tions (not shown). More interestingly, the Chern number 
is changed by —3 as we go from the C = +1 phase to the 
C = — 2 phase at a constant large B when varying the 
strength of SOC. Our analysis reveals that this transition 
can be decomposed into three steps, at each of which the 
topology of the Berry curvature distribution in the Bril- 
louin zone changes, as analyzed in the following, Fig. 0] 
(B = 1 cV). 

We focus on the neighborhood of the T-point, since 
most of the contribution to the variation of the Berry 
curvature and the Chern number exchange comes from 
the coupling of the highest occupied bands to the low- 
est unoccupied bands in this region. For £ < 0.79 £si 
in the C = +1 phase the distribution of Berry curva- 
ture exhibits a hot-loop structure with two pronounced 
singularity-like points at the k x = axis. Enhancing 
the SOC strength brings down the conduction band and 
results in a singularity directly at T-point as the bands 
touch each other. Upon reopening the gap at the touch- 
ing T-point the Chern number of the valence band is 
changed from +1 to +2 (£ = 0.80 ^Bi in Fig. 4). If the 
valence band at this £ was separated by a global gap from 
the conduction band, the Chern number of the system 
would be +2. As the £ is increased further, the conduc- 
tion band goes further down in energy, while the point 
of band crossing with the valence band and correspond- 
ing singularities in the Berry curvature split and move 
away from the T-point, resulting in a hot loop and four 
"monopoles" at £ = 0.82 ^Bi- The Chern number of the 
valence band at each of such singular points is changed 
by —1, therefore, the resulting QAH phase for £ > 82% 
of £bi has the Chern number C = -2. At 82% of the 
Bi SOC strength the band structure at the T-point is 
almost indistinguishable from that at 79%, see Fig. 4, al- 
though the Chern number, sign of the Berry curvature, 
and orbital character of valence and conduction bands 
have changed completely. 



B. Tuning £ in Bi bilayers: alloying with Sb 

We turn now to the question of how to achieve tuning 
of SOC strength and inducing a finite exchange field in 
a Bi(lll) bilayer. Tuning the strength of SOC can be 
achieved by alloying Bi with its isoelectronic but lighter 
element Sb, as demonstrated in e.g. BiTl(Si_5Sea)2 
where Se was substituted with the isoelectronic Sj^ 
In Fig. [3Jc) we show the calculated phase diagram of 
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FIG. 4: Variation of the distribution of the Berry curvature (first and third row) and the band structure (second and fourth 
row) at the T-point during a phase transition from C = +1 to C = —2 in Bi(lll) BL at B = 1 eV as a function of the SOC 
strength. Numbers below each panel indicate the percentage of the strength of SOC in unit of ^bl Black (red) lines stand for 
the bands with k y — (k x = 0). The numbers stand for the Chern numbers of the topmost valence bands. 



Sb(lll) BL with respect to the magnitude of the ex- 
change field B and the strength of SOC £. The general 
features of the phase diagram of Sb BL are similar to 
those of the Bi BL. For instance, at weak SOC strength 
and small magnetic field, both bilayers are trivial insu- 
lators, while QAH phases emerge with increasing £ and 
B. The QAH phase with Chern number C = — 2 emerges 
in Sb BL for B > 0.4 eV if the SOC strength of Sb is 
scaled by about three times of its atomic value, to reach 



£ ~ 2.7 eV, which is very close to 

Clearly, there exist also significant differences between 
the phase diagrams in Fig. 3(a) and (c). First, the TI 
phase with time-reversal symmetry at B = is reached in 
Sb BL only when the SOC strength is scaled above 305% 
of its atomic value, £ > 2.75 eV, which is larger than 
the critical value in Bi BLs £ei ~ 1.9 eV. Second, the 
boundary between the trivial insulator and the metallic 
phases is moved towards larger £ and B regime in Sb 
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BL, as compared to Bi BL. Third, and most important, 
in the Sb BL in the intermediate £ regime (0.8 ^sb < £ < 
1-8 £st>) and in a larger exchange field, the QAH phase 
bares a Chcrn number with the sign opposite to that 
in the Bi BL: C = — 1 for Sb as compared to C = +1 
for Bi. We observe that the occurrence of the C = — 1 
QAH phase is extremely sensitive to the fine details of the 
electronic structure of the bilayer. For instance, at B = 1 
cV, a "—1" QAHE phase is also present in a small region 
of the SOC strength 2.6 £ S b < k < 2.65 £ S b- Moreover, 
for 1.80 £ S b < £ < 2.60 £ sb and B«l eV, the Sb bilayer 
is gapped with a tiny gap of a few meV, which can not 
be measured in a realistic experiment in which disorder 
and temperature effects are inevitable. 

The reason for the differences in the phase diagrams 
of Sb(lll) and Bi(lll) BLs can be attributed to the dif- 
ference in the fine details of their electronic structure. 
For instance, the magnitude of the V ppa nearest neighbor 
hopping parameter in Bi accounts to 80% of that in Sb, 
while the values of the V ppa next nearest neighbor hop- 
ping parameter in Bi is only half of that in Sb, as listed 
in Ref. [30l ]. This underlines that, to a certain extent, 
Bii-^Sb^ alloys cannot be treated within a simple SOC 
strength scaling picture, and we suspect that more inter- 
esting and non-trivial topological phases might occur in 
these alloys in a strong exchange field due to the compe- 
tition between C = ±1 phases, which have to be captured 
from more accurate first principles calculations. 

The QAH phase with C=-l found in Sb BL (Fig. Etc)) 
is topologically different from the QAH phase with C=+l 
in Bi BL (Fig. EUa-b)). According to the bulk-edge cor- 
respondence, the bulk properties (Chern numbers) are 
reflected in the chirality of the edge states in a finite sys- 
tem. Obviously, the chiralities of the edge states for the 
C = ±1 QAH phases are opposite to each other, as shown 
in the insets of Fig. Eta) and Fig. [31(c). That is, the edge 
state located on the upper edge of a Sb(lll) ribbon (inset 
of Fig. Etc)) is right-propagating, while that in a Bi(lll) 
ribbon (inset of Fig. E[a)) is left-propagating. The SHC 
is of finite magnitude for both C = ±1 phases (Fig. E(b) 
for C = +1), but of opposite sign due to different chiral- 
ities of the edge states. Overall, the ±1 QAHE phases 
in Bi and Sb BLs are the quantum spin Chern insulator 
phases with opposite values of the anomalous and spin 
Hall conductivities. 



C. Tuning B in Bi bilayers: magnetic substrates 
and doping 

In order to induce a finite exchange field in Bi/Sb(lll) 
bilayers, two realistic ways can be suggested: (i) either 
by depositing the bilayers on top of a suitable magnetic 
insulating substrate whose surface layer exhibits an in- 
plane ferromagnetic spin structure, or (ii) by doping the 
systems with magnetic atoms. We first briefly consider 
(i)- 

Europium chalcogenides (EuX, X=0, S, Sc, Tc) are 



magnetic semiconductors with diverse magnetic order- 
ing and a wide range of lattice constants^ We report 
here on first principles calculations of Bi(lll) bilayer 
on top of EuSe slabs along (111) direction terminated 
with Se atoms, Fig. EJd), assuming ferromagnetic order 
in EuSe with magnetization direction perpendicular to 
the slab, but the conclusions below also hold for other 
europium chalcogenides. EuSe has cubic structure with 
the lattice constant of 6.19 A, which provides the small- 
est lattice mismatch of around 3% to Bi(lll) BL as com- 
pared to other europium chalcogenides. To describe Eu 
4/-electrons properly, we employed the LDA+J7 scheme 
with U = 7.0 eV and J = 1.2 eV. As shown in Fig. Eld), 
the states of the system around Ep are originated from 
Bi bands, with the largest exchange splitting induced by 
hybridization with the magnetic substrate accounting to 
about 60 meV. Such magnitude of the exchange split- 
ting is definitely not enough to reach the QAH phase in 
Fig. Ella), as verified by explicit AHC calculations (not 
shown), although it might be used to probe the time- 
reversal broken TI phase in the system, if the Fermi en- 
ergy of the hybrid systems can be tuned to locate in the 
gap. The small exchange splitting in the BL is due to the 
hybridization with the half-filled shell of 4/ Eu electrons, 
which is quite localized leading to insignificant exchange 
interactions in the 6p orbitals of Bi atoms. In this sense, 
using a substrate with partially filled d-shell, e.g. MnSe^ 
might be favorable. 

On the other hand, doping with or adsorption of 3d, 4d 
or 5d transition-metal atoms can induce strong exchange 
splitting and hence a non-trivial QAH effect in TIs, as 
shown in Rcfs. To demonstrate that this is in- 

deed the case for Bi bilayer, here wc consider the case 
of Fc adatoms. Compared to graphene, Bi(lll) bilayer 
has a larger in-plane lattice constant (4.52 A, as com- 
pared to 2.46 A in graphene). Our structural relaxations 
show that the Fe atoms can be stabilized at the hollow 
site of the hexagonal plaquette in the middle of two Bi 
atomic layers at the spatial inversion point of the stag- 
gered honeycomb lattice. Fig. E[e) shows the electronic 
structure of Fe adatoms in p(2 x 2) superstructure on 
Bi(lll) BL, together with the calculated anomalous Hall 
conductivity. Among two global gaps which are formed 
upon Fe deposition, the band gap at the Fermi energy 
is trivial, while the band gap at about 0.6 eV below Ep 
exhibits a non-zero Chern number C = +2 (Fig. E[ e ))- 
This is a new QAH phase, as compared to the phase di- 
agram of Bi BL in Fig. Et a ), whose emergence is due to 
a strong hybridization between the Fc d- and Bi p-states 
in the vicinity of the Fermi energy, in analogy to the case 
of graphene i 10 ' 56 A similar mechanism also plays out for 
the p(2 x 2) Co doped Bi(lll) BLs, in which case the 
QAH gap is opened at about 0.3 eV below Ep and the 
Chcrn number is +2. Furthermore, varying the strength 
of SOC on Bi atomic sites can drive the system into yet 
different topological phases. For instance, as exempli- 
fied in Fig. E[f)- if we scale the strength of SOC of Bi 
atoms to 20% of their atomic value, the gap at Ep of Fc 
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doped Bi BLs acquires non-trivial topological properties 
and displays a C = —2 QAH phase. This manifests that 
the system of 3d transition- metal adatoms/dopants on 
BiSb alloyed films definitely presents a unique and rich 
system in which exciting non-trivial phases could exist 
and could be tunable, suggesting thus the necessity for 
further theoretical and experimental investigations. 

IV. SUMMARY 

In summary, using a generic tight-binding model con- 
structed for buckled honeycomb lattice with sp-orbitals, 
we illustrated how to induce nontrivial topological phases 
by generating an effective complex next nearest neigh- 
bor hopping by spin-orbit interaction. Various channels 
can be recognized for p z -, s-, and {p x ,p y }-bands, respec- 
tively. We found that both spin and orbital degrees of 
freedom can be utilized to generate the complex hop- 
pings, and verified that for a Bi(lll) bilayer the orbital 
angular momentum purification plays an important role 
in its nontrivial properties due to the strong spin-orbit 
coupling strength of Bi atoms. Moreover, we showed how 
the nontrivial QAH phases can be induced by direct mix- 
ing of the two spin channels with corresponding exchange 
of the Chern number between the bands of opposite spin. 

Further, we demonstrated that quantum anomalous 
Hall effect in Bi(lll) bilayer can be induced by a com- 
bined tuning of the strength of the spin-orbit interaction 
and the magnitude of an external exchange field. We 
observe that various topological phases of the Bi(lll) bi- 



layer, arising when the time-reversal symmetry is broken 
can be characterized by Chern and spin Chern numbers, 
corresponding to experimentally measurable transverse 
charge and spin conductivities. We showed that tuning 
the spin-orbit strength can be achieved by alloying Bi 
with Sb, with rich physics expected in resulting Bii_ x Sb x 
systems. In order to induce a finite exchange field in a 
Bi(lll) bilayer, we examined the effect of a magnetic 
substrate, used for deposition of the bilayer, and mag- 
netic dopants. We observe that, a Bi(lll) bilayer on top 
of insulating europium chalcogenides exhibits a small ex- 
change splitting of the bands, resulting in a time-reversal 
broken topological phase. On the other hand, we predict 
that a much larger splitting can be achieved upon intro- 
ducing magnetic transition-metal impurities in the sys- 
tem, with resulting sizeable QAH gaps in the spectrum of 
the composite system formed due to strong hybridization 
between the Bi p- and adatom's d-states. The position of 
these gaps and their topological properties can be further 
tuned by tuning the spin-orbit strength of the Bi(lll) bi- 
layer via, e.g. alloying with Sb. 
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